Self-Driving Car Engineer Nanodegree¶

Deep Learning¶

Project: Build a Traffic Sign Recognition Classifier¶

In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.

Step 0: Load The Data¶

# Load pickled data
import pickle

# TODO: Fill this in based on where you saved the training and testing data

training_file = './data/traffic-signs-data/train.p'
testing_file = './data/traffic-signs-data/test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']
is_features_normal = False
is_validation_data = False
is_additional_data = False

Step 1: Dataset Summary & Exploration¶

The pickled data is a dictionary with 4 key/value pairs:

'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
'labels' is a 2D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
'sizes' is a list containing tuples, (width, height) representing the the original width and height the image.
'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below.

### Replace each question mark with the appropriate value.

# TODO: Number of training examples
n_train = len(X_train)

# TODO: Number of testing examples.
n_test = len(X_test)

# TODO: What's the shape of an traffic sign image?
image_shape = X_train.shape[1:]

# TODO: How many unique classes/labels there are in the dataset.
n_classes = len(set(y_train))

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)

Number of training examples = 39209
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.

### Data exploration visualization goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline
import random
import numpy as np
from scipy.ndimage import zoom
import os

Display one random traffic sign for each class¶

n_cols = 10
n_rows = int(np.ceil(n_classes / n_cols))
f, axarr = plt.subplots(n_rows, n_cols, figsize=(15,8))
for i in range(int(n_rows * n_cols)):
    ax = axarr[i // n_cols][i % n_cols]
    if i < n_classes:
        index = random.choice(np.where(y_train == i)[0])  #pick random item from class i
        image = X_train[index].squeeze()
        ax.imshow(image)
        ax.set_title(str(i))
        ax.axes.get_xaxis().set_visible(False)
        ax.axes.get_yaxis().set_visible(False)
    else:
        ax.axis('off')

Display first 4 traffic signs from each class¶

n_cols = 8
n_same_items = 4
n_rows = int(np.ceil(n_same_items * n_classes / n_cols))
f, axarr = plt.subplots(n_rows, n_cols, figsize=(15,40))
for i in range(int(n_rows * n_cols / n_same_items)):
    offset = i * n_same_items
    for j in range(n_same_items):
        ax = axarr[(offset+j) // n_cols][(offset+j) % n_cols]
        if i < n_classes:
                #index = random.choice(np.where(y_train == i)[0])  #pick random item from class i
                index = np.where(y_train == i)[0][j]  #pick j-th item in class i
                image = X_train[index].squeeze()
                ax.imshow(image)
                ax.set_title(str(i))
                ax.axes.get_xaxis().set_visible(False)
                ax.axes.get_yaxis().set_visible(False)
        else:
            ax.axis('off')

As shown above, almost exactly same traffic signs are shown as different images. This is because the traffic signs are taken from video stream with different time frame. So we need to be careful when split training data to training and validation set. If we randomly select validation set from the training, we will obtain unexpectedly high validation set accuracy compared to low test set accuracy.

signs, n_signs = np.unique(y_train, return_counts=True)
plt.bar(signs, n_signs)
plt.title("the number of images for each traffic sign")

<matplotlib.text.Text at 0x7f61f57569b0>

Step 2: Design and Test a Model Architecture¶

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

There are various aspects to consider when thinking about this problem:

Neural network architecture
Play around preprocessing techniques (normalization, rgb to grayscale, etc)
Number of examples per label (some have more than others).
Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

NOTE: The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!

Implementation¶

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

### Preprocess the data here.
### Feel free to use as many code cells as needed.

def normalize_image(image_data, dtype='float32'):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
    # TODO: Implement Min-Max scaling for  image data
    a = 0.1; b = 0.9
    X_min = 0; X_max = 255
    res = a + (image_data - X_min) * (b - a) / (X_max - X_min)
    return res.astype(dtype)

def clipped_zoom(img, zoom_factor, **kwargs):
    """
    Zoom-in and -out image without changing image size.
    # source http://stackoverflow.com/questions/37119071/scipy-rotate-and-zoom-an-image-without-changing-its-dimensions
    """
    from scipy.ndimage import zoom
    h, w = img.shape[:2]

    # width and height of the zoomed image
    zh = int(np.round(zoom_factor * h))
    zw = int(np.round(zoom_factor * w))

    # for multichannel images we don't want to apply the zoom factor to the RGB
    # dimension, so instead we create a tuple of zoom factors, one per array
    # dimension, with 1's for any trailing dimensions after the width and height.
    zoom_tuple = (zoom_factor,) * 2 + (1,) * (img.ndim - 2)

    # zooming out
    if zoom_factor < 1:
        # bounding box of the clip region within the output array
        top = (h - zh) // 2
        left = (w - zw) // 2
        # zero-padding
        out = np.zeros_like(img)
        out[top:top+zh, left:left+zw] = zoom(img, zoom_tuple, **kwargs)

    # zooming in
    elif zoom_factor > 1:
        # bounding box of the clip region within the input array
        top = (zh - h) // 2
        left = (zw - w) // 2
        out = zoom(img[top:top+zh, left:left+zw], zoom_tuple, **kwargs)
        # `out` might still be slightly larger than `img` due to rounding, so
        # trim off any extra pixels at the edges
        trim_top = ((out.shape[0] - h) // 2)
        trim_left = ((out.shape[1] - w) // 2)
        out = out[trim_top:trim_top+h, trim_left:trim_left+w]

    # if zoom_factor == 1, just return the input array
    else:
        out = img
    return out

from scipy.ndimage import shift
from scipy.ndimage import rotate

# perturb image to genarate additional data
def perturb_image(img, delta_pixel_range=(-2,2), scale_ratio_range=(0.9,1.1), rotation_range=(-15,15)):
    """
    Shift (translate), scale (Zoom-in and -out), and roate　input images randomly within specified ranges.
    """
    # translation
    dx = random.randint(*delta_pixel_range)
    dy = random.randint(*delta_pixel_range)
    shifted_img = shift(img, (dx,dy,0))

    # scaling
    dscale = random.uniform(*scale_ratio_range)
    zoomed_img = clipped_zoom(shifted_img, dscale)
    
    # rotation
    dangle = random.uniform(*rotation_range)
    res = rotate(zoomed_img, dangle, reshape=False)
    return res

Question 1¶

Describe how you preprocessed the data. Why did you choose that technique?

Answer:
The feature data for training, validation and test data are normalized by Min-Max scaling to a range of [0.1, 0.9] to improve the convergence of gradient decent. The data normalization was intentionally conducted after checkpoint to reduce disk size below, where the data is save as 'uint8' data type (before normalization) instead of 'float32' (after normalization), where the saving data in 'uint8' type significantly reduce data size. (About 4 times less)

### Generate data additional data (OPTIONAL!)
### and split the data into training/validation/testing sets here.
### Feel free to use as many code cells as needed.

# Split training datasets for training and validation in order to minimize the mixture of traning and validation data
if not is_validation_data:
    train_indeces = np.array([]).astype('uint8')
    valid_indeces = np.array([]).astype('uint8')
    for i in range(n_classes):
        indeces = np.where(y_train == i)[0] # select indeces belong to class i
        index_thres = int(len(indeces) * 0.95) # choose 95% as threshold
        train_indeces = np.concatenate([train_indeces, indeces[:index_thres]]) # first 95% in eath class as training
        valid_indeces = np.concatenate([valid_indeces, indeces[index_thres:]]) # last 5% as validaiton
    X_train, X_validation = X_train[train_indeces], X_train[valid_indeces]
    y_train, y_validation = y_train[train_indeces], y_train[valid_indeces]
    is_validation_data = True
len(X_train), len(X_validation), len(X_test)

(37239, 1970, 12630)

sample pertubed image¶

image1 = X_train[400].squeeze()
fig, ax = plt.subplots(1, 2)
ax[0].imshow(image1)
ax[1].imshow(perturb_image(image1))

/home/carnd/anaconda3/lib/python3.5/site-packages/scipy/ndimage/interpolation.py:568: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
  "the returned array has changed.", UserWarning)

<matplotlib.image.AxesImage at 0x7f61f559bbe0>

# Split training datasets for training and validation in order to minimize the mixture of traning and validation data
if not is_additional_data:
    n_new_data_for_each_image = 4
    X_train_new = []
    y_train_new = []
    for index in range(len(X_train)):
        # original data
        X_train_new.append(X_train[index])
        y_train_new.append(y_train[index])
        # additional data
        for i in range(n_new_data_for_each_image):
            X_train_new.append(perturb_image(X_train[index]))
            y_train_new.append(y_train[index])
    X_train = np.array(X_train_new)
    y_train = np.array(y_train_new) 
    del X_train_new; del y_train_new
    is_additional_data = True
    
len(X_train)

/home/carnd/anaconda3/lib/python3.5/site-packages/scipy/ndimage/interpolation.py:568: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
  "the returned array has changed.", UserWarning)

186195

X_train.shape, y_train.shape

((186195, 32, 32, 3), (186195,))

# Save the data for easy access
# we'll save the data before normalization to save space (after normalization dtype for X_train will be float32 from uint8 and data size increases)
import os
pickle_file = 'data/traffic-signs-data_train_validation_2.pickle'
if not os.path.isfile(pickle_file):
    print('Saving data to pickle file...')
    try:
        with open(pickle_file, 'wb') as pfile:
            pickle.dump(
                {
                    'X_train': X_train,
                    'y_train': y_train,
                    'X_validation': X_validation,
                    'y_validation': y_validation,
                },
                pfile, pickle.HIGHEST_PROTOCOL)
    except Exception as e:
        print('Unable to save data to', pickle_file, ':', e)
        raise

print('Data cached in pickle file.')

Saving data to pickle file...
Data cached in pickle file.

sample pertubed images¶

n_cols = 8
n_same_items = 4
n_rows = int(np.ceil(n_same_items * n_classes / n_cols))
f, axarr = plt.subplots(n_rows, n_cols, figsize=(15,40))
for i in range(int(n_rows * n_cols / n_same_items)):
    offset = i * n_same_items
    for j in range(n_same_items):
        ax = axarr[(offset+j) // n_cols][(offset+j) % n_cols]
        if i < n_classes:
                #index = random.choice(np.where(y_train == i)[0])  #pick random item from class i
                index = np.where(y_train == i)[0][j]  #pick j-th item in class i
                image = X_train[index].squeeze()
                ax.imshow(image)
                ax.set_title(str(i))
                ax.axes.get_xaxis().set_visible(False)
                ax.axes.get_yaxis().set_visible(False)
        else:
            ax.axis('off')

# the training data set increased 5 times more than before

signs, n_signs = np.unique(y_train, return_counts=True)
plt.bar(signs, n_signs)
plt.title("the number of images for each traffic sign")

<matplotlib.text.Text at 0x7f61c57096d8>

Checkpoint¶

We can start from here, once new training and validation data are generated and saved.

# Load pickled data
import pickle

training_validation_file = './data/traffic-signs-data_train_validation_2.pickle'
testing_file = './data/traffic-signs-data/test.p'

with open(training_validation_file, mode='rb') as f:
    train_validation = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train_validation['X_train'], train_validation['y_train']
X_validation, y_validation = train_validation['X_validation'], train_validation['y_validation']
X_test, y_test = test['features'], test['labels']
del train_validation
is_features_normal = False
is_validation_data = True
is_additional_data = True

if not is_features_normal:
    X_train = normalize_image(X_train)
    X_validation = normalize_image(X_validation)
    X_test = normalize_image(X_test)
    is_features_normal = True

Question 2¶

Describe how you set up the training, validation and testing data for your model. Optional: If you generated additional data, how did you generate the data? Why did you generate the data? What are the differences in the new dataset (with generated data) from the original dataset?

Answer:
First, I split the training data to training and validation sets. This was carefully conducted because if one selects validation data randomly, many of validation data would be very similar to training set due to the sequential similarity of original images. To avoid this I deliberately split the original training data so that the first 95% to be new training set and last 5% to be validation set for each class. The test data is kept untouched.
Secondly, I generated 4 additional perturbed data for each of training data, and then the number of training set became 5 times more than the original. As result the test set accuracy improved to 94% from 91% with naive LeNet-5. The perturbed images are created by adding (i) translation to a range of [-2, 2] pixel, (ii) scaling to range of [0.9, 1.1] times, and rotation to range of [-15, 15] degree randomly to original image for 4 times.
Then, new dataset is saved as the pickle format to create checkpoint.
After that, before we train the model, the feature data for training, validation, and test set are normalized by Min-Max scaling to a range of [0.1, 0.9].

### Define your architecture here.
### Feel free to use as many code cells as needed.

import tensorflow as tf
import math
from tqdm import tqdm
from sklearn.utils import shuffle

### Implement LeNet-5
from tensorflow.contrib.layers import flatten

def LeNet(x):    
    # Hyperparameters
    mu = 0
    sigma = 0.1
    
    #from IPython.core.debugger import Tracer; Tracer()()

    # Layer 1: Convolutional. Input = 32x32x3. Output = 28x28x6.
    W1 = tf.Variable(tf.truncated_normal([5, 5, 3, 6], mean=mu, stddev=sigma))
    b1 = tf.Variable(tf.zeros(6))
    layer1 = tf.nn.bias_add(tf.nn.conv2d(x, W1, strides=[1, 1, 1, 1], padding='VALID'), b1)

    # Activation.
    layer1 = tf.nn.relu(layer1)

    # Pooling. Input = 28x28x6. Output = 14x14x6.
    layer1 = tf.nn.max_pool(layer1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

    # Layer 2: Convolutional. Output = 10x10x16.
    W2 = tf.Variable(tf.truncated_normal([5, 5, 6, 16], mean=mu, stddev=sigma))
    b2 = tf.Variable(tf.zeros(16))
    layer2 = tf.nn.bias_add(tf.nn.conv2d(layer1, W2, strides=[1, 1, 1, 1], padding='VALID'),
                                             b2)
    # Activation.
    layer2 = tf.nn.relu(layer2)
    # Pooling. Input = 10x10x16. Output = 5x5x16.
    layer2 = tf.nn.max_pool(layer2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
    # Flatten. Input = 5x5x16. Output = 400.
    layer2 = flatten(layer2)
    
    # Layer 3: Fully Connected. Input = 400. Output = 120.
    W3 = tf.Variable(tf.truncated_normal([400, 120], mean=mu, stddev=sigma))
    b3 = tf.Variable(tf.zeros(120))
    layer3 = tf.add(tf.matmul(layer2, W3), b3)    
    # Activation.
    layer3 = tf.nn.relu(layer3)
    # Dropout
    layer3 = tf.nn.dropout(layer3, keep_prob)

    
    # Layer 4: Fully Connected. Input = 120. Output = 84.
    W4 = tf.Variable(tf.truncated_normal([120, 84], mean=mu, stddev=sigma))
    b4 = tf.Variable(tf.zeros(84))
    layer4 = tf.add(tf.matmul(layer3, W4), b4)
    # Activation.
    layer4 = tf.nn.relu(layer4)
    # Dropout
    layer4 = tf.nn.dropout(layer4, keep_prob)

    # Layer 5: Fully Connected. Input = 84. Output = n_classes.
    W5 = tf.Variable(tf.truncated_normal([84, n_classes], mean=mu, stddev=sigma))
    b5 = tf.Variable(tf.zeros(n_classes))
    logits = tf.add(tf.matmul(layer4, W5), b5)    

    return logits

tf.reset_default_graph()

x = tf.placeholder(tf.float32, (None, 32, 32, 3))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, n_classes)
keep_prob = tf.placeholder(tf.float32) # probability to keep units

rate = 0.001

logits = LeNet(x)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, one_hot_y)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss_operation)

correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()

def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: 1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

Question 3¶

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.

Answer:
Type of model is use:

# LeNet-5. with dropout

Layers, sizes, connectivity:

# Layer 1: Convolutional. Input = 32x32x3. Output = 28x28x6.
# Activation: Relu
# Pooling. Input = 28x28x6. Output = 14x14x6.

# Layer 2: Convolutional. Output = 10x10x16.
# Activation: Relu
# Pooling. Input = 10x10x16. Output = 5x5x16.
# Flatten. Input = 5x5x16. Output = 400.

# Layer 3: Fully Connected. Input = 400. Output = 120.
# Activation: Relu

# Layer 4: Fully Connected. Input = 120. Output = 84.    
# Activation: Relu
# Dropout: Keep 50% 

# Layer 5: Fully Connected. Input = 84. Output = 43.
# Dropout: Keep 50%

### Train your model here.
### Feel free to use as many code cells as needed.

len(X_train), len(y_train)

(186195, 186195)

EPOCHS = 100
BATCH_SIZE = 64
KEEP_PROB = 0.5

# Measurements use for graphing loss and accuracy
log_epoch_step = 2
epochs = []
loss_epoch = []
train_acc_epoch = []
valid_acc_epoch = []

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train)
    batch_count = int(math.ceil(num_examples / BATCH_SIZE))
    l = float('inf')
    training_accuracy = 0
    validation_accuracy = 0
    
    for i in range(EPOCHS):
        X_train, y_train= shuffle(X_train, y_train)
        # Progress bar
        batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(i+1, EPOCHS), unit='batches')
        
        for batch_i in batches_pbar:
            offset = batch_i * BATCH_SIZE
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset:end], y_train[offset:end]
            _, l = sess.run([training_operation, loss_operation], feed_dict={x: batch_x, y: batch_y, 
                                                                             keep_prob: KEEP_PROB})
            
        # Log every log_epoch_step batches
        if not i % log_epoch_step:
            # Calculate Training and Validation accuracy
            training_accuracy = evaluate(X_train, y_train)
            validation_accuracy = evaluate(X_validation, y_validation)

            # Log epochs
            previous_epoch = epochs[-1] if epochs else 0
            epochs.append(log_epoch_step + previous_epoch)
            loss_epoch.append(l)
            train_acc_epoch.append(training_accuracy)
            valid_acc_epoch.append(validation_accuracy)

            print("EPOCH {} ...".format(i+1), flush=True)
            #print("Loss = {:.3f}".format(l), flush=True)
            print("Training Accuracy = {:.3f}".format(training_accuracy), flush=True)
            print("Validation Accuracy = {:.3f}".format(validation_accuracy), flush=True)
            fname = 'data/check_points/'
            saver.save(sess, os.path.join(fname, 'lenet'), global_step=(i+1))
            print("Model saved", flush=True)        

    training_accuracy = evaluate(X_train, y_train)
    validation_accuracy = evaluate(X_validation, y_validation)
    print("Training Accuracy = {:.3f}".format(training_accuracy))
    print("Validation Accuracy = {:.3f}".format(validation_accuracy))
    saver.save(sess, 'lenet')
    print("Model saved")

Epoch  1/100: 100%|██████████| 2910/2910 [00:19<00:00, 148.84batches/s]:   0%|          | 12/2910 [00:00<00:25, 113.15batches/s]

EPOCH 1 ...
Training Accuracy = 0.766
Validation Accuracy = 0.837
Model saved

Epoch  2/100: 100%|██████████| 2910/2910 [00:19<00:00, 148.92batches/s]:   1%|          | 15/2910 [00:00<00:20, 142.40batches/s]
Epoch  3/100: 100%|██████████| 2910/2910 [00:19<00:00, 148.54batches/s]

EPOCH 3 ...
Training Accuracy = 0.903
Validation Accuracy = 0.948
Model saved

Epoch  4/100: 100%|██████████| 2910/2910 [00:19<00:00, 148.87batches/s]:   1%|          | 15/2910 [00:00<00:20, 142.80batches/s]
Epoch  5/100: 100%|██████████| 2910/2910 [00:19<00:00, 149.03batches/s]

EPOCH 5 ...
Training Accuracy = 0.935
Validation Accuracy = 0.970
Model saved

Epoch  6/100: 100%|██████████| 2910/2910 [00:19<00:00, 148.90batches/s]:   1%|          | 15/2910 [00:00<00:20, 140.74batches/s]
Epoch  7/100: 100%|██████████| 2910/2910 [00:19<00:00, 148.74batches/s]

EPOCH 7 ...
Training Accuracy = 0.947
Validation Accuracy = 0.974
Model saved

Epoch  8/100: 100%|██████████| 2910/2910 [00:19<00:00, 148.82batches/s]:   0%|          | 14/2910 [00:00<00:20, 139.47batches/s]
Epoch  9/100: 100%|██████████| 2910/2910 [00:19<00:00, 148.91batches/s]

EPOCH 9 ...
Training Accuracy = 0.955
Validation Accuracy = 0.980
Model saved

# Adopted result.  Dropout keep_prob = .5 and epoch = 100
# Even though it looks underfit, it peformed best for test set accucary. 
# This is likely because of the similary between training set and validation set.
loss_plot = plt.subplot(211)
loss_plot.set_title('Loss')
loss_plot.plot(epochs, loss_epoch, 'g')
loss_plot.set_ylim([0.0, 0.5])
loss_plot.set_xlim([epochs[0], epochs[-1]])
acc_plot = plt.subplot(212)
acc_plot.set_title('Accuracy')
acc_plot.plot(epochs, train_acc_epoch, 'r', label='Training Accuracy')
acc_plot.plot(epochs, valid_acc_epoch, 'x', label='Validation Accuracy')
acc_plot.set_ylim([0.9, 1.0])
acc_plot.set_xlim([epochs[0], epochs[-1]])
acc_plot.legend(loc=4)
plt.tight_layout()
plt.show()
print("Training Accuracy = {:.3f}".format(training_accuracy), flush=True)
print("Validation Accuracy = {:.3f}".format(validation_accuracy), flush=True)

Training Accuracy = 0.986
Validation Accuracy = 0.991

# Accuracy result with test set
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    test_accuracy = evaluate(X_test, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))

Test Accuracy = 0.963

Question 4¶

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:
Optimizer: AdamOptimizer (Becauase it is known that Adam works very well for default optimizer when intensive parameter turning is not conducted).
Learning rate: 0.001(Other learning rates such as 0.05, 0.005, etc. are also tried and 0.001 turns out to be bast).
Batch Size: 64 (Here 32, 124, and 248 are also tried, and 64 worked best).
Epochs: 100 (Training and Validation accuracies are visualized and the number of epochs is decided when they are stabled).
Dropout Keep prob: 0.5 (Here, 0.6 and 0.7 are also tried and it turns out that 0.5 works best.

Question 5¶

What approach did you take in coming up with a solution to this problem? It may have been a process of trial and error, in which case, outline the steps you took to get to the final solution and why you chose those steps. Perhaps your solution involved an already well known implementation or architecture. In this case, discuss why you think this is suitable for the current problem.

Answer:
Data preparation:
First I carefully split the original training set to generate the validation to observe if the implemented model is over-fit or under-fit at the model training.
Then, I created the additional data by randomly perturbed original data 4 times, so that model can adapt more variety of images, such as slightly tilted or shifted images than original ones. The size of training data set becomes 5 times as many as original.
In addition, the feature data (X_data, X_validation, X_test) is normalized to help gradient descent work better.

Initial architecture selection:
I selected the well known but relatively simple LeNet-5 architecture for the starting point.

Parameter turning 1:
First I restricted the training data and epochs to small so that I can quickly identify the good learning rate and batch size for the architecture.
Then, I extended the data size and epochs to obtain the best training and validation accuracies with the parameters.

Architecture modification: Obtaining the above training and validation accuracies, I notice that training accuracy is much better than validation accuracy, indicating model is over-fitting.
Therefore, I added dropout to the 4th- and 5th-layers (fully connected layers), to regularize the model.

Parameter turning 2: Now, since I had one more parameter to turn, i.e. dropout keep prob. I tried 0.5, 0.6, and 0.7, and then found 0.5 is the best, and extend the number of epochs to 100.

History of test accuracy improvements:
Naive LeNet-5 without additional training data (epochs: 60): 90.6%
Naive LeNet-5 with additional training data (epochs: 60): 93.9%
Naive LeNet-5 with additional training data and dropout (epochs: 60, keep_prob: 0.5): 96.0%
Naive LeNet-5 with additional training data and dropout (epochs: 100, keep_prob: 0.5): 96.3%

Step 3: Test a Model on New Images¶

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Implementation¶

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

### Load the images and plot them here.
### Feel free to use as many code cells as needed.

# load, resize, and normalize input data.
# As the input data, the images of local traffic sign in Japan are used.
from scipy.misc import imresize
dname = 'data/japan_signs/'
name_sample = [f for f in os.listdir(dname) if f.endswith("jpg")]
X_sample = [plt.imread(os.path.join(dname, f)) for f in name_sample]
X_tmp = []
for i in X_sample:
    X_tmp.append(imresize(i, (32, 32)))  # resize images to (32, 32, 3)
X_sample = np.array(X_tmp)
X_sample = normalize_image(X_sample)
name_sample = np.array(name_sample)

X_sample.shape, name_sample.shape

((8, 32, 32, 3), (8,))

n_cols = 4
n_items = len(X_sample)
n_rows = int(np.ceil(n_items / n_cols))
f, axarr = plt.subplots(n_rows, n_cols, figsize=(10,5))
for i in range(int(n_rows * n_cols)):
    ax = axarr[i//n_cols][i%n_cols]
    if i < n_items:
            image = X_sample[i].squeeze()
            ax.imshow(image)
            ax.set_title(name_sample[i].split('.')[0])
            ax.axes.get_xaxis().set_visible(False)
            ax.axes.get_yaxis().set_visible(False)
    else:
        ax.axis('off')

Question 6¶

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It could be helpful to plot the images in the notebook.

Answer:
8 Japanese local traffic signs shonwn above are selected and the qualities of the images are described below:

"ahead_only" (Japanese):
"35:Ahead only" (German) is almost identical both semantically and pictorially.
dangerous_curve_to_left (Japanese)
There is no similar traffic sign in German. "19:Dangerous curve to the left" (German) is very distinct view.
no_entry (Japanese)
"17:No entry " (German) is almost identical both semantically and pictorially.
no_truck (Japanese)
There is no similar traffic sign in German. "10:No passing for vehicles over 3.5 metric tons" (German) is kind of similar but distinct by the direction of truck and red line from upper left to lower right.
no_vehicles (Japanese)
There is no similar traffic sign in German. "15:No vehicles" (German) is very distinct by the figure of car in the center and red line from upper left to lower right.
slippery (Japanese)
There is no similar traffic sign in German. "30:Beware of ice/snow" (German) may be closest by semantically but figure is very distinct.
speed_limit_50km (Japanese)
"2:Speed limit (50km/h)" (German) is almost identical both semantically and pictorially.
stop (Japanese)
"14:Stop" (German) is almost identical semantically, but pictorially there are quite distinct. Although both signs have red background and white characters, the shape of German sign is octagon and the of Japan is triangle, and also the language written on the German sign is English while the one on Japanese is Japanese.

### Run the predictions here.
### Feel free to use as many code cells as needed.

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    sess = tf.get_default_session()
    softmax1 = sess.run(tf.nn.softmax(logits), feed_dict={x: X_sample, keep_prob:1.0})
    predictions = sess.run(tf.argmax(tf.nn.softmax(logits), 1), 
                           feed_dict={x: X_sample, keep_prob:1.0})

predictions

array([35, 36, 17,  7,  5, 20,  2, 14])

import pandas as pd
file1 = 'signnames.csv'
signnames = pd.read_csv(file1, index_col=0)

n_cols = 2
n_items = len(X_sample)
n_rows = int(np.ceil(n_items / n_cols))
f, axarr = plt.subplots(n_rows, n_cols * 2, figsize=(11,13))
for i in range(int(n_rows * n_cols)):
    ax = axarr[i // n_cols][2*(i % n_cols)]
    if i < n_items:
        image = X_sample[i].squeeze()
        ax.imshow(image)
        title = "Left: Local Sign (Japan)\n" + name_sample[i].split('.')[0]
        ax.set_title(title)
        ax.axes.get_xaxis().set_visible(False)
        ax.axes.get_yaxis().set_visible(False)
    else:
        ax.axis('off')
    ax = axarr[i // n_cols][2*(i % n_cols) + 1]
    if i < n_items:
        pred_index = predictions[i]
        index = random.choice(np.where(y_train == pred_index)[0])  #pick random item from class i
        image = X_train[index].squeeze()
        ax.imshow(image)
        title = "Right: Predected (Germany)\n" + str(pred_index) + ":"+ signnames.iloc[pred_index, 0]
        ax.set_title(title)
        ax.axes.get_xaxis().set_visible(False)
        ax.axes.get_yaxis().set_visible(False)
    else:
        ax.axis('off')

Question 7¶

Is your model able to perform equally well on captured pictures when compared to testing on the dataset? The simplest way to do this check the accuracy of the predictions. For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate.

NOTE: You could check the accuracy manually by using signnames.csv (same directory). This file has a mapping from the class id (0-42) to the corresponding sign name. So, you could take the class id the model outputs, lookup the name in signnames.csv and see if it matches the sign from the image.

Answer:
4 out of 8 Japanese local traffic signs are predicted correctly. Therefore the accuracy is 50%. Considering miss classified 4 signs have very distinct figure in Germany, the classifier is working very well.
One very surprising result is that the Japanese stop sign is correctly identified by Germany traffic sign classifier though there shape and the letter on sign is very distinct.

### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    sess = tf.get_default_session()
    softmax1 = sess.run(tf.nn.softmax(logits), feed_dict={x: X_sample, keep_prob:1.0})
    res = sess.run(tf.nn.top_k(softmax1, k=5))

# For each  Japanese traffic sign, top 5 German traffic sign forecasts are given with certainty (above figures)
n_cols = k = 5
n_items = len(X_sample)
n_rows = n_items
f, axarr = plt.subplots(n_rows * 2, n_cols, figsize=(15,55))
for i in range(n_items):
    for j in range(n_cols):
        ax = axarr[2*i][j]
        if j == 0:
            image = X_sample[i].squeeze()
            ax.imshow(image)
            title = "#{} Input: {} (Japan)".format(i+1, name_sample[i].split('.')[0] )
            ax.set_title(title)
            ax.axes.get_xaxis().set_visible(False)
            ax.axes.get_yaxis().set_visible(False)
        else:
            ax.axis('off')
        ax = axarr[2*i +1][j]
        pred_index = res.indices[i,j]
        index = random.choice(np.where(y_train == pred_index)[0])  #pick random item from class i
        image = X_train[index].squeeze()
        ax.imshow(image)
        title = "#{} Prediction {}: {:.1f}%\n {}: {}".format(i+1, j+1, res.values[i,j] * 100, res.indices[i,j], signnames.iloc[res.indices[i,j], 0])
        ax.set_title(title)
        ax.axes.get_xaxis().set_visible(False)
        ax.axes.get_yaxis().set_visible(False)

Question 8¶

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.

Take this numpy array as an example:

# (5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])

Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:

TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))

Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.

Answer:
Interestingly, the 4 of 8 correctly predicted signs have the certainty of 100.0% for top-1 predictions. Regarding the rest of 4 incorrectly predicted signs, theier certainties of top-1 predictions varg from 28.8% to 79.0%.
Looking at the 4 incorrectly predicted signs closely, 3 of 4 could not be predicted correctly in top-5. The last one, slippery (Japanese), was predicted correctly as 5th candidates with the certainty of 5.3%. Again, since the incorrectly predicted Japanese signs do not have similar signs　in Germany, their low accuracy is reasonable.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.